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Background / Context: 

Description of prior research and its intellectual context. 

Increasing postsecondary academic success is a national imperative. While college enrollments 
have increased about 300 percent from just over 5.9 million students in 1965 to about 17.5 
million students in 2005 (Snyder, Dillow, and Hoffman, 2008), college graduation rates have 
been fairly stagnant over the same period (Eaton, 1997; Turner 2004). As a result, the U.S. is 
losing ground compared to other nations in the percentage of the adult population attaining 
college degrees (Organisation for Economic Co-operation and Development, 2007). 

Financial aid may improve access to and persistence in college. Previous research suggests that 
financial aid is positively associated with increased enrollment (St. John et al., 2002; Kane 2004; 
Dynarski 2000, 2003; Cornwell, Mustard, and Sridhar, 2006) and positively associated with 
increased persistence (Leslie & Brinkman, 1987; St. John et al., 1991, 1994, 2001; Cabrera, 

Nora, and Castaneda, 1993; Choy, 2002; DesJardins, Ahlburg, and McCall, 2002; Bettinger, 
2004; Singell and Stater, 2006; Scott-Clayton, 2009). 

In contrast to the literature cited above, there are a handful of studies on the effectiveness of 
financial incentives for postsecondary students that have employed random assignment designs. 
Angrist, Lang, & Oreopoulos (2009) study a financial incentive with academic services offered 
to first-year, traditional students at a Canadian university. At the end of their second year, 
women who received both money and support services earned higher grades than women who 
received neither, but there were no impacts on the academic performance of men. Goldrick-Rab, 
Harris, Benson & Kelchen (201 1) study a need-based grant offered to first-time Wisconsin 
students enrolling in a Wisconsin public institution. The additional financial aid resembled that 
of the longstanding federal Pell Grant program. An analysis of the first cohort finds that while 
there is no impact on persistence, credits attempted or completed, students in the program group 
were almost 30 percent more likely to have earned at least 60 credits after two years. 

Brock and Richburg-Hayes (2006), Richburg-Hayes et al. (2009), and Barrow et al., (2009) 
evaluated a financial incentive offered to nontraditional community college students as part of 
MDRC’s Opening Doors Demonstration in Louisiana. The program targeted low-income 
parents, who were predominantly single mothers. The early findings from the Opening Doors 
program resulted in substantial improvements in scholarship recipients’ grades, credit 
accumulation, and persistence (Brock & Richburg-Hayes, 2006; Richburg-Hayes et al., 2009). 
For example, 65 percent of the program group registered for courses in the second semester, 
compared with 50 percent of the control group (a 30 percent increase). 

Although the Opening Doors results were strong, one study is not sufficient to verify the 
effectiveness of a program. Additionally, the devastation inflicted by Hurricane Katrina has 
made it difficult to confirm the long-term program effects. The Performance-Based Scholarship 
(PBS) Demonstration is a multi-state, multi-design efficacy and replication study intended to test 
the effectiveness of performance-based scholarships. Each site targets a different low-income 
population with high financial need that is not fully covered by existing financial aid programs. 
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Table 1 presents a matrix illustrating the design in each of the sites (please insert Table 1 here). 
We propose to share findings from three of the earliest sites. 

Purpose / Objective / Research Question / Focus of Study: 

Description of the focus of the research. 

The intention of the program is that conditioning additional financial aid on certain behaviors 
will lead students to increase their effort towards their studies, which in turn will lead them to 
perform better in their classes in the short-term and progress through their degree requirements at 
a quicker rate in the medium-term. The program is hypothesized to result in increased enrollment 
in subsequent terms, increased credits attempted and earned, and increased financial aid. 

Increases in these short-term education outcomes may then lead to changes in medium- and long- 
term educational outcomes, including year-to-year persistence, total cumulative credits earned, 
as well as graduation or transfer to another postsecondary institution. The proposed presentation 
will focus on the primary research question: What is the impact of performance-based 
scholarships (in various settings with different types of students) on short-term academic 
outcomes as well as medium- and long-term academic outcomes? We will present current 
findings on these outcomes, across the six sites in our demonstration. 

Setting: 

Description of the research location. 

The demonstration is operating in six key states: New York, Ohio, New Mexico, California, 
Arizona and Florida. Table 1 details the participating institutions and organizations in each state. 

Population / Participants / Subjects: 

Description of the participants in the study: who, how many, key features, or characteristics. 

The demonstration targets different populations in each state, in order to determine the impact of 
these scholarships across different target groups and in different geographic areas. The general 
eligibility requirements require that all study participants are low-income. More detailed 
eligibility information is provided in Table 1. 

Intervention / Program / Practice: 

Description of the intervention, program, or practice, including details of administration and duration. 

The performance-based scholarship program differs in each state, and is described in detail in 
Table 1. While each state is a different test, all of the programs encompass the following: 

• Scholarships are paid to students based on their academic performance in the current 
term, regardless of what happened in previous terms (or in the case of many merit-based 
scholarships, what happened in high school). 

• Scholarships are paid directly to students rather than credited to their accounts with the 
college. That way, students can use the money for their most pressing needs. 

• Scholarships supplement Pell grants and state aid to help meet the needs of low-income 
students. The scholarships are designed to be paid on top of existing federal and state aid. 
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Research Design: 

Description of the research design. 

The evaluation employs a random assignment research design. Students who met program 
eligibility criteria at each site were randomly assigned to two groups: a program group that 
received the performance-based scholarship, or a control group that received whatever financial 
aid was available to all students. Within the program group in New York only, students were 
randomly selected to be eligible to receive an additional summer term scholarship through being 
assigned to a summer-eligible group. Within the program group in California only, students were 
randomly assigned to one of the six different scholarship types, varying in duration and amount. 
Random assignment ensured that the motivation levels and personal characteristics of students in 
all groups were the same at the start of the study. By tracking the groups over time, researchers 
could measure the difference, or impact, that the study had on student outcomes. 

Data Collection and Analysis: 

Description of the methods for collecting and analyzing data. 

The analysis will be based on data collected from several data sources: 

• Baseline Information Form, consisting of demographic and other background information 
on students prior to any influence by the program 

• Student-level academic records, used to provide a look at sample members’ performance 
in college through measures such as enrollment, credits attempted and earned, grades, 
degrees earned, and financial aid receipt. Data are collected from the colleges 
themselves, state organizations, and the National Student Clearinghouse. 1 

• Scholarship payment data on the Performance-Based Scholarship payments 

• Survey data based on surveys conducted in Ohio and California in the first semester 
following random assignment. The surveys include data on education and time-use, 
employment, motivation and experience, program participation and perceptions, and 
health and well-being. 

Findings / Results: 

Description of the main findings with specific details. 

We plan to share findings that are currently available for Ohio, New York, and New Mexico. 
Findings for California, Arizona and Florida are not currently available but may be by the 
conference date. 

Ohio 

We analyzed outcomes from the first year of the program using student transcript data from the 
Ohio Board of Regents for the fall 2008 cohort. This sample is about 1,300 students (60 percent 
of the total sample). As shown in Table 2, the program increased the number of regular credits 
attempted and earned over the program year and increased the proportion of students earning 24 
or more credits by 6.6 percentage points (an increase of more than 42 percent). The program also 
increased the total amount of financial aid reported by the program group and decreased the 
amount of loans in their financial aid package. 



1 The National Student Clearinghouse collects and distributes enrollment, degree, and certificate data from 3,300 
colleges that enroll 93 percent of the nation's college students (National Student Clearinghouse, 2011). 
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New York 

To estimate early impacts, we analyzed outcomes from the first semester and first summer term 
during which the program operated using student transcript data from the colleges. The first two 
cohorts. Fall 2008 and Spring 2009, were analyzed (60 percent of the full sample). The early 
findings suggest that the program encouraged slightly more students to enroll full time rather 
than part time and resulted in a small number of students registering for summer courses when 
eligible for additional summer funding. 

New Mexico 

To estimate early impacts, we analyzed student transcript outcomes for the full sample over the 
first two semesters during which the program operated. As shown in Table 3, the early findings 
suggest that the primary effect of the program is a large increase in the number of students 
attempting at least 15 credits during the second semester. This indicates that the program is 
helping a larger proportion of students to stay on track to graduate in four years. In addition, the 
program resulted in a reduction in loans (as in Ohio above). 

California 

California is currently the most ambitious program in the demonstration because it offers 
different versions of scholarships that are portable - i.e., can be used at any accredited college or 
university in the country. The goal is to measure the effectiveness of varying scholarship 
amounts for varying durations in order to help determine whether there is a tipping point at 
which the scholarship would have the greatest impact. The PBS Demonstration randomly 
awarded six versions of performance-based scholarships that range from $500 to $1,000 per 
term, and that last between one term and two years. 

At this point, we have preliminary scholarship take-up rates and matriculation information for 
the sample. Table 4 shows that about 82 percent of program group students provided the required 
information to receive their first scholarship payment. Furthermore, over half of program group 
students who received their first payment matriculated at public 2-year colleges, 42 percent 
matriculated at public 4-year colleges, and 96 percent of students remained in California. 

Conclusions: 

Description of conclusions, recommendations, and limitations based on findings. 

The PBS Demonstration is really the first of its kind, testing out different geographical locations, 
with different amounts of monies, over different durations to ascertain the best means of 
implementing such a scholarship. The results of the Opening Doors Louisiana study had 
prompted the state of Ohio to try the performance-based scholarship model (Cha & Patel, 2010). 
It is our hope that the results of this study can inform an honest and much-needed discussion on 
need-based aid for a variety of low-income populations across the country to increase their 
academic success and possibly change their life trajectories. 

Since the geographic location, target populations and program designs differ across each state, 
differences in impacts cannot be attributed directly to any of these varying components. This 
diversity was intentional in the design of the Demonstration in order to reach a wide variety of 
student populations and test a number of different designs. 
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Appendix B. Tables and Figures 

Not included in page count. 

Table 1 



Sites and Sample Criteria 





New York City 


Ohio 


University of New 
Mexico 


California 


Arizona 


Florida 


Sample Selection Criteria 


Age 


22-35 


At least 1 8 


16-26 

(primarily 17-19) 


16-19 


No age criteria 


At least 1 8 


Low-Income 

Threshold 


Pell-Eligible 


EFC of 0 2 


Pell-Eligible 


Below Cal Grant A/C 
Threshold 


EFC below 5,273 s 


EFC below 5,273 s 


Additional 

Requirements 


Live away from 
parents 

In need of at least 1 
remedial course 


Must be a parent 


Must be 1 st year 
student 


Must be 1 st year 
student 

Must have completed 
FAFSA and Cal Grant 


Hispanic males 
< 45 credits earned 


In need of remedial 
math 


Intervention 


Scholarship 
amount 
per term 


$1,300 


$300 - $900 (part-time 
and full-time amounts 
vary) 


$1,000 


$333 -$1000 
(6 program groups 
with varying amounts 
and durations) 


Up to $1,500 (part- 
time and full-time 
amounts vary by 
academic / service 
criteria) 


$600 


Scholarship 

duration 


2 semesters plus 1 
summer semester 3 


2 semesters / 3 
quarters 


4 semesters 


1-4 semesters / 1-6 
quarters 


3 semesters 


3 semesters + 1 
summer term 


Scholarship 

distribution 


$200 at enrollment 
$450 at midterm 
$650 at term end 


Full payment 
at term end 


$250 at enrollment 
$250 at midterm 
$500 at term end 


Varies by amount and 
duration 


$150 at orientation 
$150 at advising 
Balance at term end 


$100 at enrollment 
$500 at term end 


Maximum total 
scholarship amt 


$2,600 to $3,900 2 


$1,800 


$4,000 


$4,000 


$4,500 


$1,800 


Portable? 


No 


No 


No 


Yes 


No 


No 


Cohorts of study 


Fall 2008 
Spring 2009 
Fall 2009 


Fall 2008 
Winter 2009 
Spring 2009 


Fall 2008 
Fall 2009 


Fall 2009 
Fall 2010 


Fall 2010 
Spring 2011 
Fall 2011 


Fall 2010 
Spring 20 1 1 
Fall 2011 



2 The EFC (Expected Family Contribution) is the amount of money that a family is expected to be able to contribute to a student’s education, as calculated 
according to federal guidelines. Students with an EFC of up to 5,273 during the 2010-201 1 academic term are eligible for federal Pell grants. 

3 New York had a 3-way design where half of all New York program group students are eligible to receive a summer scholarship. 
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New York City 


Ohio 


University of New 
Mexico 


California 


Arizona 


Florida 


Years of program 
operation 


Fall 2008 to Summer 
2010 


Fall 2008 to Winter 
2010 


Fall 2008 to Spring 
2011 


Fall 2009 to Spring 
2012 


Fall 2010 to Fall 2012 


Fall 2010 to Fall 2012 


Performance Benchmarks 


Academic criteria 


Enroll in college, and 
complete at least 6 
credits with a “C” in 
each 


Part time: 6-11 credits 
with “C” or better in 
each 

Full time: 12 or more 
credits, “C” or better 
in each 


Complete at least 12 
(first semester) or 15 
(subsequent 
semesters) credits with 
a “C” or better average 


Enroll in college, and 
complete at least 6 
credits with a “C” or 
better average 


Part time: 6-11 credits 
with “C” or better in 
each 

Full time: 12 or more 
credits, “C” or better 
in each 


Complete a sequence 
of math courses with a 
“C” or better 


Service criteria 


N/A 


N/A 


Meet with advisor 


N/A 


Meet with advisor, 
complete tutoring and 
workshop 
requirements 


Complete tutoring 
requirements 


Form of payment 


Check or direct 
deposit 


Check 


Check 


Direct deposit (85%+) 
or check 


Check 


Payment on HCC One 
card (student card) 


Experimental Design 


Research Design 


3-way design 


2-way design 


2-way design 


6-way design 


2-way design 


2-way design 


RA Ratio (P:C) 


25:25:50 


60:40 


50:50 


33:67 


60:40 


64:36 


Total number of 
students in study 
& program group 


1,502 total 
754 program group 


2,285 total 
1,359 program group 


1,081 total 
536 program group 


5,160 total 
1,720 program group 


1 ,000 total 
600 program group 
(projected) 


1,050 total 
650 program group 
(projected) 


Participating Institutions 


Participating 

Institutions 


Hostos Community 
College 

Borough of Manhattan 
Community College 


Lorain County 
Community College 

Owens Community 
College 

Sinclair Community 
College 


University of New 
Mexico 


California Student Aid 
Commission and LA 
Chamber of 
Commerce operate 
Cash for College 
workshops and the 
PBS program. 
Student can take 
scholarship to any 
accredited 2-4 yr 
institutions in US 


Pima Community 
College 


Hillsborough 
Community College 
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New York City 


Ohio 


University of New 
Mexico 


California 


Arizona 


Florida 


Funders and 
Partners 


Robin Hood 
Foundation 

NYC Center for 
Economic Opportunity 


The Ohio Department 
of Job and Family 
Services through the 
Ohio Board of 
Regents 

The Joyce Foundation 
IES 


Open Society 
Foundation 

IES 


College Access 
Foundation of 
California 

IES 


Kresge Foundation 

Helios Foundation 

Open Society 
Foundation 


Helios Foundation 

Open Society 
Foundation 
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CONFIDENTIAL DO NOT CITE OR DISTRIBUTE 

Table 2 

Early Findings for the Fall 2008 Cohort in Ohio 



Lorain County, Owens, and Sinclair Community Colleges 



Outcome 


Program 

Gtoup 


Control 

Gtoup 


Difference 


Standard 

Error 


Academic outcomes - Full nroszram year 


Registered for any courses (%) 


98.3 


97.8 


0.5 


0.8 


Number of credits attempted 3 


20.0 


19.5 


0.5 


0.4 


Regular credits 


17.7 


16.9 


0.8 * 


0.4 


Developmental credits 


2.4 


2.6 


-0.2 


0.2 


Number of credits earned 3 


15.3 


13.4 


2 o *** 


0.5 


Regular credits 


13.9 


12.0 


j g *** 


0.5 


Developmental credits 


1.4 


1.4 


0.1 


0.2 


Earned 24 or more credits 3 (%) 


22.0 


15.4 


6.6 *** 


2.2 


Earned 12 to 23 credits 3 (%) 


45.0 


45.0 


-0.1 


2.8 


Sample size (n = 1,329) 


782 


547 






Financial aid awarded - First nroszram term ' 1 


Total financial assistance awarded ($) 


4,504 


4,164 


340 *** 


92 


Federal Pell grant 


1,699 


1,754 


-55 


33 


Performance-Based Scholarship 


676 


0 


676 *** 


9 


Other grants 3 


654 


659 


-5 


27 


Subsidized loans 


779 


901 


-122 *** 


39 


Unsubsidized loans 


650 


765 


-115 ** 


47 


Federal work study 


45 


84 


-39 * 


22 


Sample size (n = 1,214) 


720 


494 







SOURCE: MDRC calculations from Ohio Board of Regents transcript data forthe Fall 2008 cohort, and MDRC 
calculations from financialaid data provided by Lorain County, Owens, and Sinclair Community Colleges. 

NOTES: Rounding may cause slight discrepancies in sums and differences. 

A two-tailed t-test was applied to differences between research groups. Statistical significance levels are 
indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Estimates are adjusted by campus. 

The program year is considered the fall 2008 and spring 2009 semesters for Lorain County and Owens 
Community Colleges, and the fall 2008, winter 2009 and spring 2009 quarters for Sinclair Community College. 
a These are based on semester-equivalent credits. 

b The financialaid impacts exclude students for whom MDRC did not recieve financialaid data at the 
midterm point (29 students: 14 program group, 15 control group). It further excludes Lorain students whose 
financialaid was repackaged before their initial financialaid data could be collected (86 students: 48 program 
group, 38 control group). 
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Table 3 

Early Findings for the First Two Semesters in New Mexico 



University of New Mexico 





Program 


Control 




Standard 


Outcome 


Qoup 


Qoup 


Difference 


Error 


Academic outcomes - Full sample 










First Droeram semester 










Registered for any courses (%) 


98.0 


99.0 


-0.9 


0.7 


Number of credits attempted 


14.8 


15.0 


-0.2 


0.2 


Regular credits 


13.2 


13.5 


-0.2 


0.2 


Developmental credits 


1.6 


1.6 


0.0 


0.1 


Attempted 15 or more credits (%) 


73.2 


75.8 


-2.6 


2.7 


Second Droeram semester 










Registered for any courses (%) 


91.8 


91.4 


0.4 


1.7 


Number of credits attempted 


14.1 


13.3 


08 *** 


0.3 


Regular credits 


13.8 


13.0 


0 7 *** 


0.3 


Developmental credits 


0.4 


0.3 


0.1 


0.1 


Attempted 15 or more credits (%) 


80.4 


55.8 


24.5 *** 


2.7 


Sample size (n = 1,081) 


536 


545 






Financial aid disbursement - Fall 2008 cohort only 










First Droeram semester 










Total financial assistance disbursed ($) 


5,885 


5,328 


557 *** 


204 


Federal Pell grant 


1,711 


1,786 


-75 


64 


NM Bridge to Success or Lottery grant 


1,093 


1,105 


-11 


55 


VISTA Scholarship 


853 


0 


853 *** 


19 


Other grants" 


1,544 


1,448 


96 


160 


Subsidized loans 


349 


546 


_197 *** 


69 


Unsubsidized loans 


205 


214 


-9 


75 


Federal work study 


130 


229 


_99 ** 


50 


Second Droeram semester 










Total financial assistance disbursed ($) 


5,057 


5,001 


56 


218 


Federal Pell grant 


1,592 


1,685 


-93 


73 


NM Bridge to Success or Lottery grant 


1,431 


1,556 


-125 


85 


VISTA Scholarship 


622 


0 


622 *** 


27 


Other grants" 


867 


816 


50 


106 


Subsidized loans 


259 


519 


-260 *** 


65 


Unsubsidized loans 


193 


219 


-26 


74 


Federal work study 


93 


205 


-111 ** 


43 


Sample size (n = 487) 


240 


247 







SOURCE: MDRC calculations from University of New Mexico transcript and financial aid data. 

NOTES: Rounding may cause slight discrepancies in sums and differences. 

A two-tailed t-test was applied to differences between research groups. Statistical significance levels are 
indicatedas: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Academic impact estimates are adjusted by the following student characteristics: gender, race/ethnicity, 
mother and father's education levels, current employment, language spoken at home, high school GPA, and 
ACT composite score. 

■'This includes all grants and scholarships excluding the Pell Grant, NM Bridge to Success and Legislative 
Lottery Grant, and VISTA scholarships. 
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Scholarship Receipt for the First Program Term 
California 



Outcome 


Program 

Group 


First oroaram term 


Claimed scholarship (%) 


93.1 


Received a scholarship payment (%) 


81.5 


Received initial enrollment payment 


81.5 


Received other payment a 


0.1 


Average scholarship amount received ($) 


475 


Average scholarship among recipients 
Among those who received a scholarship payment (n = 1,401): 


583 


Breakdown by sector (%) 


Public 2-year 


51.5 


Public 4-year or above 


42.2 


California State University 


26.1 


University of California 


14.4 


Private 2-year 


0.2 


Private 4-year or above 


6.1 


Breakdown by state (%) 


In-state ( CA) 


96.4 


Out-of-state 


3.6 


Breakdown by term type (%) 


Semester-based system 


78.1 


Quarter-based system 


21.9 


Credits attempted on enrollment verification 


12.9 


Semester-equivalent credits attempted on enrollment verification b 


11.9 


Enrolled fidl-time at institution of enrollment verification c (%) 


80.2 


Enrolled part-time at institution of enrollment verification d (%) 


19.8 


Payment made by electronic fund transfer (%) 


88.2 


Payment made by check (%) 


11.5 


Sample Size 


1,720 



SOURCE: MDRC calculations from scholarship payment data provided by the California Student Aid 
Commission (CSAC) aswell as data from U.S. Department of Education, NationalCenterfor Education 
Statistics, Integrated Postsecondary Education Data System (IPEDS). The data presented in this table may 
change slightly as students can still submit verification and receive a payment forthe periods presented. 

NOTES: Characteristics shown in italics are calculated fora subset of the program group. 

Institution information shown is based on data provided with the initial enrollment payment forthe first 
program term. 

The institution information presented is for a student's main institution of attendance, but credits 
attempted and earned represent the total across multiple institutions if more than one is attended. 

“Other payments indicate payment amounts that did not line up with the scheduled payment amount for 
the student's scholarship type for a given semester or quarter institution. We are following up with CSAC to 
clarify these payments. 

b Credits attempted at quarterinstitutions have been multiplied by 2/3 to obtain semester -equivalency. 

“Full-time enrollment is defined as 12 or more credits, without adjustment for quarter institutions. 

d Part -time enrollment is defined as 6 to less than 12 credits, without adjustment forquarterinstitutions. 
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